The present invention relates to the field of color image displays, and more particularly to color displays in which the color image planes are separately updated or displayed. The invention also relates to compensation of motion images for artifacts introduced by a discrete time representation thereof.
Color image displays are of two general types. In a first type, exemplified by a typical direct-view cathode ray tube color display, all color image components are displayed simultaneously. Thus, an image model, e.g., a CCIR-601 signal, defines the luminance and chrominance of each image pixel at a particular time. The motion image is therefore presented as a time sequence of color image frames.
In a second type of color image display, color image planes are displayed sequentially. This type of system is employed, for example, in certain single panel image projection systems, in which light of various colors sequentially illuminates a common spatial light modulator. The spatial image modulator, therefore, modulates the intensity of each respective color component of a pixel sequentially and independently, which is perceived as a color motion image.
Sequential color displays work well as long as one pixel is being observed during an entire frame time. When a moving object is present, the eyes are focussing on this object, and the eyes start tracking the motion. The color breakup artifact that an observer notices at that moment is caused due to tracking of the motion by the eyes, i.e. the eyes follow a moving object by rotating the head and eyes while keeping this object focussed on the same position on the retina.
Color sequential displays display the Red, Green, Blue (RGB) colors alternating during a frame time, as represented in FIG. 1. This frame time might have a small delay for each successive row, depending on the way the color sequential illumination is implemented, but this time delay is generally considered negligible. The image pixels are observed at different time moments within a frame period, and thus might display different video data for the RGB colors other than the intended one. If this happens when the video data of these pixels changes, at that moment a color break-up artifact is visible.
JP 08-123355 A, published May 17, 1996, relates to a motion compensation system for a plasma display panel. In this system, the image motion artifact caused by the tracking of the human eye of motion of a displayed object between successive frames, in a display system having pulse modulated gray scale generation, representing a plurality of image subframes differing in brightness, is corrected by calculating object motion and moving the object within the image for correction at the time of display.
The following references are hereby incorporated herein by reference in their entirety:
W. Bruls, A. van der Werf, R. Kleihorst, T. Friedrich, E. Salomons and F. Jorritsma, xe2x80x9cA single-chip MPEG-2 encoder for consumer storage applicationsxe2x80x9d Digest of the ICCE, 1997, Chicago, pp. 262-263.
G. de Haan and H. Huijgen, xe2x80x9cMotion estimation for TV picture enhancementxe2x80x9d, in Signal Processing of HDTV III, (H. Yasuda and L. Chiariglione, eds.), Elseviers Science Publishers B.V., 1992, pp. 241-248.
G. de Haan, J. Kettenis, and B. Deloore, xe2x80x9cIC for motion compensated 100 Hz TV, with a smooth motion movie-modexe2x80x9d, IEEE Transactions on Consumer Electronics, vol. 42, May 1996, pp. 165-174.
G. de Haan, P.W.A.C. Biezen and O. A. Ojo, xe2x80x9cAn Evolutionary Architecture for Motion-Compensated 100 Hz Television,xe2x80x9d in IEEE Transactions on Circuits and Systems for Video Technology, Vol. 5, No. 3, June 1995, pages 207-217.
xe2x80x9cMotionxe2x80x94compensated picture signal interpolationxe2x80x9d, U.S. Pat. No. 5,495,300, Inventors: G. de Haan, P. Biezen, A. Ojo, and H. Huijgen.
G. de Haan, P. Biezen, H. Huijgen, and O. Ojo, xe2x80x9cTrue motion estimation with 3-D recursive search block-matching,xe2x80x9d IEEE Transactions on Circuits and Systems for Video Technology, vol. 3, No. 5, October 1993, pp. 368-388.
G. de Haan, P.W.A.C. Biezen, H. Huijgen and O. A. Ojo, xe2x80x9cGraceful Degradation in Motion-Compensated Field-Rate Conversionxe2x80x9d in Proceedings of the International Workshop on HDTV, Ottawa, Canada, 1993, pages 249-256.
JP 06-46358, 18.2.1994, xe2x80x9cLiquid Crystal Driving Devicexe2x80x9d, Masao Kawamura. (Moving image emphasis system).
WO 96/35294, xe2x80x9cMotion Compensated Filteringxe2x80x9d, 7 Nov. 1996, Timothy Borer (Motion compensated filtering of interlace video). See also U.S. Pat. Nos. 5,335,194; 4,862,266; and 4,862,267;
The present inventors have therefore determined that the perceived artifacts evident in a color sequential display producing an image of a moving object may be addressed by separately motion compensating the objects represented within the respective color planes based on the time of display. This compensation scheme determines motion vectors for areas or objects within an image stream, predicts (e.g., by interpolating or extrapolating) the object position for respective times of presentation of respective color subframes, and sequentially displays the respective color planes representing the areas or objects at the predicted position.
Artifacts may be evident any time a dynamic pixel image is displayed at a time other than the theoretical timing of presentation. Thus, while the artifact is especially apparent in color sequential RGB displays driven using the nominal RGB color separation of a sequence of composite frames of a video signal, it may also occur in certain other instances, for example in systems employing other color plane definitions or color space representations. Therefore, it is understood that the invention is not limited to the modification of color plane images of RGB sequential color displays to correct for the non-coincidence of the actual time of display with the theoretical time of display with respect to the motion of objects within a frame. In fact, the invention encompasses the modification of image subframe data for display, especially where the image subframes represent different components of the image frame and are presented at different respective times than the nominal frame time, by reformulating the discrete time image subframes by distinguishing image object components having apparent independent motion, and resynthesizing an image subframe with a modified relationship object of the respective object components having independent motion. Preferably, this resynthesis is for the purpose of correcting a timing shift between the actual time of display of the image subframe including the object component and the theoretical time for display based on the original scene. The present invention also provides a system and method for calculating and presenting the image corrections with sub-pixel precision.
The prediction of object or area position may be of any known type, although interpolation or extrapolation of first order (velocity) is preferred. Higher order predictions, for example, require additional information, and may-therefore increase image presentation latency and/or memory requirements. While known motion estimation techniques, such as those employed in MPEG-2 encoders, may be employed, model based systems may also be employed, for example MPEG-4 or VRML-type systems, to define the object motion.
Motion estimation systems typically operate on image frames to analyze sequential changes. Where analogous regions are identifiable between sequential frames, typically confined to a search area, the displacement of the region may be encoded as a motion vector. Therefore, this information is often used in motion image compression, wherein the identification of an image block and its motion vector is transmitted, instead of the image block itself. According to the present invention, similar techniques may be used, as is well known in the art. However, portions of a moving foreground object may cover or uncover the background image. In the case of uncovering of background, the background image need be synthesized. In the case where an image interpolation is employed, the background image may be predicted based on the subsequent image frame. In the case of an extrapolation (i.e., use of past frame data only), this information may be lacking. Therefore, in spite of the use of intelligent algorithms, artifacts may be generated in this case as well.
According to the present invention, it is important to encode the motion vectors to represent the movement of objects represented in the image, with priority given to the largest motion vectors, rather than the greatest opportunity for image compression.
The implementation of a processor for computing motion vectors is well known in the art, and need not be described herein in detail. Typically, the motion vector compensation system will include a powerful general purpose computer or workstation, or a dedicated digital image processing device, such as an appropriately programmed Royal Philips Electronics Trimedia device.
Motion estimation analyzes the speed of the video frame content and the motion vectors are determined. Motion compensation, on the other hand, uses these motion vectors to re-align (or interpolate) the three RGB colors according to these motion vectors. In FIG. 2, the motion artifact and the result of motion compensation are shown. These artifacts occur generally in sequential color display systems, although an analogous artifact exists in Plasma Display Panels (PDPs) and Digital Mirror Devices (DMDs).
According to the present invention, in color field compensation, the color fields are aligned on the motion vectors (with the remaining rounding errors). In color sequential displays, the colors are aligned on the motion vectors with a remaining rounding error, or more accurately with the preferred bilinear interpolation technique. This is possible in color sequential displays, since the entire amplitude for one color is being displayed in one color subframe, in contrast to color field (PDP) gray scaling. According to an aspect of the invention, bit splitting techniques, frame rate increasing techniques, and motion compensation may be simultaneously employed, by controlling each subfield and each color plane individually. For example, when using traditional bit splitting of the most significant bit (MSB), when making a correction of ne-half of the MSB subfields, the other half MSB subfield is also xe2x80x9cfalselyxe2x80x9d corrected. According to the present invention, two bits are available for controlling each half of the MSB subfield individually.
Therefore, it is an object according to the present invention to provide a system and method for image compensation in an image display device in order to avoid various motion image artifacts, especially image color break-up and improve a perceived imaged quality.
It is a further object of the invention to provide a color sequential image display processor which reduces color break-up by estimating an optimal position of an image representation of a moving object or area based on a time of presentation and predicted or estimated motion characteristics.
It is also an object according to the present invention to provide a system for motion compensation of color sequential displays.
Such a system comprises means for estimating motion from the input image data; means for calculating a motion compensated luminance value from the input video data, valid at a temporal instance that corresponds to a gravity center of each color subframe; and means for calculating the value for all color subframes leading to correct perception of the motion for the combination of the color subframes. The calculation of the correct color subframe video value is preferably done by bilinear interpolation; alternately, the calculation of the correct color subframe video value is done by selecting the closest pixel to choose the video data. The motion compensation means preferably includes a robust interpolation method reducing the effect of erroneous motion vectors, the robust interpolation method being selected from one or more of median filtering and order statistical filtering. Advantageously, a decision of a reference time for the most important color subframe, e.g., green, is taken at a moment in time for which no motion compensation interpolation, or compensation over the shortest possible distance from the original picture, is required.
It is also an object according to the present invention to provide a method for motion compensation of color sequential displays.
Such a method comprises the steps of:
(a) processing received image data comprising frames (2), each frame defining a plurality of subframes (4, R, G, B), each subframe representing a different component of the image frame (R, G, B) for display at different respective times (TR, TG, TB) within a frame period;
(b) estimating a motion of an image portion represented in said frames of image data input (3); and
(c) motion compensating the image portion based on the estimated motion, with respect to a respective time instance of display of at least one of said subframes thereof (5), thereby reducing display artifacts.
These and other objects and features of the present invention will become more fully apparent from the following description and appended claims taken in conjunction with the accompanying drawings.